The present invention relates to a composite display apparatus including an image-pickup optical system for guiding light from the image information of its external world to an image-pickup device and a display optical system for guiding light exiting from a liquid crystal display etc. to the eyeballs of an observer.
There have been proposed units including a head mounted display (HMD) and a miniature CCD camera attached thereto and displaying image information obtained from the camera on a display device included in the HMD directly or after subjecting the image information to some image processing. There have also been proposed various see-through type HMDs which use optical coupling/separating devices, such as a half mirror, and enable not only the observation of images displayed on a display device, but also the direct observation of its external world at an angular magnification of xc3x971.
In the above forms of HMDs, a difference between the ocular optical axis of the display optical system, which is for allowing the person equipped with the HMD to observe the image information displayed on the display device of the HMD, and the optical axis of the CCD camera (image-pickup optical system), which is for allowing the image information of its external world to form an image on the CCD surface, results in parallax.
So there have been proposed units including both image-pickup optical system and display optical system which is free from parallax in Japanese Patent Application Laid-Open No. 4-22358 and Japanese Patent Application Laid-Open No. 5-303053.
The display optical system of the visual aid unit proposed in Japanese Patent Application Laid-Open No. 4-22358 is constructed in such a manner that light emitted from a miniature liquid crystal display unit passes through a plane half mirror and then is guided to eyeballs by an ocular. On the other hand, the image-pickup optical system of the same is constructed in such a manner that light of its external world passes through the plane half mirror and then forms an image in a miniature camera which is provided in such a position as to face the liquid crystal device across the half mirror. This visual aid unit, however, tends to be large because the plane half mirror used in the unit is arranged at an angle of 45 degrees. In addition, this unit cannot apply to see-thorough type units because there is arranged an ocular in front of the eyeballs of an observer.
On the other hand, in Japanese Patent Application Laid-Open No. 5-303053, there is disclosed a head mounted display system which includes an image pickup optical system and a display optical system and is capable of constituting a see-through type unit; however, the unit has a complicated optical system and tends to be very large.
The applicants of the present invention have already overcome these problems and proposed composite display apparatus in Japanese Patent Application Laid-Open No. 11-174367 which includes a display optical system with more than one eccentric reflection surface for guiding a light beam from a display means, such as a liquid crystal display (LCD), to the eyeballs of an observer and an image-pickup optical system for allowing image information of its external world to form an image on the surface of an image-pickup device, such as CCD, and enable preventing the light beam from the display means from entering the pickup surface of the image-pickup optical system, in addition, observing the image information of both optical systems and creating an image of the same in a good state where there results no parallax while allowing the entire unit to be miniaturized.
FIG. 10 is a schematic view showing the main part of one form of the composite display apparatus proposed in Japanese Patent Application Laid-Open No. 11-174367. In the same figure, reference numeral 101 denotes a display optical system (LCD display optical system) which consists of a prism body 9a utilizing more than one internal reflection, includes more than one eccentric reflection surface each having more than one eccentric curvature, and realizes a color display using a ferroelectric display device 2 (FLCD), as a display means, by illuminating the display device with three light sources 31R, 31G and 31B radiating respective color light in a time-sharing manner one by one while synchronizing the illumination with the display of the color image separated into colors R, G and B. R, G and B images are displayed within {fraction (1/60)} to {fraction (1/30)} seconds altogether utilizing the afterimage effect. Each of the light sources 31R, 31G and 31B can be constructed by combining a white-light source and a color filter. The image information displayed on the FLCD 2 is guided to the eyeballs of an observer through a display optical system 101. The display optical system 101a is an optical axis.
Reference numeral 102 denotes an image-pickup optical system (CCD image-pickup optical system) which, like the display optical system 101, consists of a prism body 9c utilizing more than one internal reflection, includes more than one eccentric reflection surface each having more than one eccentric curvature, and images the image information of its external world on an image-pickup device (CCD) 3. Reference numeral 102a is an optical axis of the image-pickup optical system. The display optical system 101 and the image-pickup optical system 102 are joined together via an optical path separating means 4.
The optical path separating means 4 consists of eccentrically curved reflection surfaces, which separate the optical paths from the display optical system 101 and from the image-pickup optical system 102. The optical path separating means 4 consists of eccentric reflection surfaces, the surface on one side being used in the display optical system 101 and the surface on the other side being used in the image-pickup optical system 102.
The light beam from the image information displayed on the FLCD 2 exits in the form of linearly polarized light. PP denotes a sheet polarizer and its polarization axis are established in such a direction as to be at right angle to the polarization axis of the linearly polarized light radiated from the FLCD 2. Thus, the linearly polarized light from the FLCD 2 is prevented from entering a CCD 3 through an optical system 6 via the optical path separating means 4 and prism body 9c, as described later. Reference numeral 5 denotes a diaphragm. Numeral 6 denotes an optical system which images the image of its external world entering via the prism body 9c on a CCD 3 surface.
Further, this composite display apparatus allows its external world to be visible through the prism bodies 9a, 9c; accordingly, it can constitute a see-through type unit. In other words, the users of this composite display apparatus observe an image which is the superposition of image information of its external world and that displayed on the FLCD 2. When establishing a see-through optical system 103, in addition to the display optical system 101 and the image-pickup optical system 102, for observing image information of its external world alone, the reflection surface of the optical path separating means 4 is a half mirror (the transmittance-to-reflectance ratio is not limited to 50%:50%, but is arbitrary) and the refractive index of the materials of the prism bodies 9a, 9c of both optical systems 101, 102 are the same. Reference numeral 1c denotes an optical axis of the see-through optical system 103 (external world optical axis).
In FIG. 10, the optical axis 101a of the display optical system 101, the optical axis 102a of the image-pickup optical system 102 and the optical axis 1c of the see-through optical system 103 coincide at the optical path separating means 4 and become co-axial with an ocular optical axis 1a. Establishing each constituent as above enables a miniaturized optical system. Further, establishing the refracting power of each eccentric reflection surface differently depending on the azimuth angle around its vertex enables maintaining a satisfactory optical performance.
In such a composite display apparatus, the light from the FLCD 2 enters an incidence surface E1 of the prism body 9a which constitutes the display optical system 101, reflects on a reflection surface 8a, enters a total reflection surface 7a at an incidence angle more than critical angle and totally reflects thereon, reflects on the optical path separating means 4, enters the total reflection surface 7a at an incidence angle less than critical angle and penetrates a penetration surface E2, and is guided to the eyeballs 1 of an observer. The display optical system reflects the light beam from the FLCD 2 twice or three times without creating an image and guides the same to the eyeballs 1 of an observer, as described above. Thus the image information displayed on the FLCD 2 is presented to the observer.
On the other hand, the light from the external world of the unit enters an incidence surface E3 (part of a total reflection surface 7c) of the prism body 9c which constitutes the image-pickup optical system 102, reflects on the optical path separating means 4, enters the total reflection surface 7c at an incidence angle more than critical angle and totally reflects thereon, exits from an exit surface E4 of the prism body 9c, and then passes through the diaphragm 5 and the optical system 6, so as to create an image on the CCD 3. Thus the image information of the external world can be obtained.
In the see-through optical system 103 shown in FIG. 10, the light from the external world enters the incidence surface E3 of the prism body 9c of the image-pickup optical system, penetrates the optical path separating means 4 (half mirror), enters the exit surface E2 at an incidence angle less than critical angle, and penetrates the exit surface E2, so as to be guided to the eyeballs 1 of an observer. Thus, the observer can observe the image information of the external world.
In FIG. 10, the sheet polarizer PP is arranged in the neighborhood of the diaphragm 5 of the image-pickup optical system 102, that is, in the neighborhood of the position of the observer""s pupils. The polarization axis of this sheet polarizer PP is established in such a manner as to be at right angles to that of the linearly polarized light from the FLCD 2.
Thus, the light from the FLCD 2 is prevented from entering the CCD 3 surface through the optical system 6 via the optical path separating means 4 and prism body 9c and from causing flare, thereby the optical properties of the image information of the external world obtained on the CCD 3 can be maintained satisfactorily.
The total reflection surface 7a and the exit surface E2 of the prism body 9a consist of the same curved surface, and they are changed properly depending on the incidence conditions of the light beam. The incidence surface E3 and the total reflection surface 7c of the prism body 9c also consist of the same curved surface, and they are changed properly depending on the incidence conditions of the light beam, like the case of the prism body 9a. 
In the construction of this display unit, the ocular optical axis 1a of the light beam entering the eyeballs 1 or an imaginary ocular optical axis 1b which is an extension of the ocular optical axis 1a, the external world optical axis 1c of the light beam entering the image-pickup optical system 102 from the external world (the optical axis 102a of the image-pickup optical system 102) are allowed to almost coincide by arranging the display optical system 101 and the image-pickup optical system 102 facing each other across the optical path separating means 4, thereby the observation of the image information displayed on the FLCD 2, imaging of the image information of the external world on the CCD 3 surface and observation of the image information of the external world (see-through optical system) are enabled simultaneously in state where there results no parallax.
FIG. 11 shows the same composite display apparatus as shown in FIG. 10, except that it includes a color LCD 40 with a backlight 46 as a display means.
In FIG. 11, the color liquid crystal display LCD 40 includes a sheet polarizer 41, a cover glass 42, a color filter 43 consisting of R, G and B micro filters arranged two-dimensionally at a predetermined pitch, a liquid crystal display device 44, and a sheet polarizer 45.
The color image information (pixels) displayed on the liquid crystal display device 44 is illuminated with the light beam (white light) from the backlight 46 via the sheet polarizer 45 and the light beam from each pixel is allowed to exit via the color filter 43 and the sheet polarizer 41.
In the construction as above, the same effects as the construction shown in FIG. 10 can be produced by establishing the polarization axis of the polarized light beam exiting from the color liquid crystal display LCD 40 in such a manner as to be at right angles to the polarization axis of the sheet polarizer PP. The other constituents and construction are the same as that of FIG. 10, therefore the description thereof shall be omitted here.
FIG. 8 shows one example of the circuit configurations of the currently used composite display apparatus, FIG. 9A one example of the timing charts of the composite display apparatus having the construction shown in FIG. 10, and FIG. 9B one example of the timing charts of the composite display apparatus having the construction shown in FIG. 11.
In FIG. 8, reference numeral 1001 denotes a system control portion for controlling the entire composite display apparatus, which consists of a computer system including CPU, ROM, RAM, etc. Numeral 1003 denotes a CCD control portion for performing timing control over the CCD 3, adjustments of image-pickup data, etc., numeral 1004 an image-pickup system interface portion for converting image-pickup data to a data format which can be input to the system controlling portion 1001, numeral 1005 a display system interface portion for converting display data output from the system control portion 1001 to a data format which can be displayed by the FLCD 2 (or the color LCD 40), numeral 1006 an LCD control portion for performing timing control over the FLCD 2 (or the color LCD 40), adjustments of the display data, etc., and numeral 1011 an illumination light source control portion for controlling operation timing of illumination light sources (31R, 31G, 31B or 46).
In FIGS. 9A and 9B, xe2x80x9cIMAGE-PICKUP DATAxe2x80x9d shows the state of image-pickup data output from the CCD 3, and xe2x80x9cSHUTTERxe2x80x9d the state of an electronic shutter, High indicating the closed state, Low indicating the open state. xe2x80x9cDISPLAY DATAxe2x80x9d shows the state of the contents displayed on the FLCD 2 (or the color LCD 40); xe2x80x9cILLUMINATION LIGHT SOURCE Rxe2x80x9d, xe2x80x9cILLUMINATION LIGHT SOURCE Gxe2x80x9d and xe2x80x9cILLUMINATION LIGHT SOURCE Bxe2x80x9d in FIG. 9A show the states of the red illumination light source (31R), green illumination light source (31G) and blue illumination light source (31B), respectively; and xe2x80x9cILLUMINATION LIGHT SOURCExe2x80x9d in FIG. 9B shows the sate of the white illumination light source 46. In each illumination state, High indicates the operating state and Low indicates non-operating state.
In the following, the examples of the currently used composite display apparatus will be further described.
In the image-pickup system, image-pickup data are output from the CCD 3 based on the timing controlled by the CCD control portion 1003, subjected to data adjustments, such as CDS (Correlated Double Sampling), AGC (Automatic Gain Control) and AWB (Automatic White Balance), at the CCD control portion 1003, converted to standard data of, for example, NTSC and PAL, at the image-pickup system interface portion 1004, and introduced into the system control portion 1001.
On the other hand, in the display system, display data of, for example, VGA and LDI formats output from the system control portion 1001 are converted at the display system interface portion 1005 to a data format which can be displayed by the LCD (2 or 40), subjected to data adjustments such as a gamma correction at the LCD control portion 1006, and output to the LCD (2 or 40) based on the timing generated so as to display an image thereon. In this process, the signal of the display timing is input to the illumination light source control portion 1011 as well, so that the operation timing of the illumination light source (31 or 46) can be controlled according to the display timing.
In order for the operation timing of the illumination light source (31 or 46) in the display system not to coincide with the timing of opening the electronic shutter in the image-pickup system, first the display timing signal of the LCD control portion 1006 is input to the CCD control portion 1003 to establish synchronization of the display system and the image-pickup system. Then, the illumination light source (31 or 46) is allowed to be in the on state for a fixed period of time by the illumination light source control portion 1011, and a trigger signal for opening the electronic shutter for a fixed period time is output from the CCD control portion 1003 to the CCD 3.
At this point, the shutter speed is generally set at {fraction (1/100)} seconds in view of the occurrence of flicker when using the display unit indoors. And based on 1 field (60 Hz) of the NTSC, which is a standard signal of a video system, the period of one cycle is {fraction (1/60)} seconds, accordingly the operating duration of the illumination light source (31 or 46) at this time is ({fraction (1/60)}-{fraction (1/100)}) seconds or less, that is, 6 milliseconds.
In the aforementioned composite display apparatus proposed in Japanese Patent Application Laid-Open No. 11-174367, however, no particular reference is made to the adjustments of the quantity of the image display light and the shutter speed of the CCD camera, but they are treated as fixed to some extent.
Accordingly, when using the composite display apparatus in a light place, if the quantity of light from the display means, such as a liquid crystal display (LCD), is kept constant regardless of the increase in quantity of the see-through light guided from the external world directly to the eyes of an observer, the display tends to be faint relative to the see-through images, and if the shutter speed of the image-pickup device, such as a CCD, is kept constant, flare and smear may be caused on the picked-up image due to the excess light quantity.
Contrarily, when using the composite display apparatus in a dark place, if the quantity of light from the display means, such as a liquid crystal display (LCD), is kept constant regardless of the decrease in quantity of the see-through light guided from the external world directly to the eyes of an observer, the display tends to be strong relative to the see-through images, and if the shutter speed of the image-pickup device, such as a CCD, is kept constant, the picked-up image may be dark or the contrast thereof may be decreased due to the insufficient light quantity.
The present invention has been made in light of the problems attendant to the prior arts described above; accordingly, one of the objects of the present invention is to provide a composite display apparatus which enables keeping both optical density of display images formed by a display means and shutter speed of a CCD etc. in an optimum state relative to the light from its external world (see-through light), even in the use environments in which it is assumed that the quantity of see-through light varies widely.
Another object of the present invention is to provide a flexible composite display apparatus by allowing a composite display apparatus to be free from the constraint that the field period of the image-pickup system should correspond to the frame period of the display system.
Specifically, the composite display apparatus of the present invention includes a display optical system for guiding a light beam from a display means to the eyeballs of an observer; an image-pickup optical system for allowing a light beam from its external world to form an image on an image-pickup means; and an optical path separating means provided in optical path for allowing an ocular optical axis of the light beam of the display optical system entering the eyeballs of the observer, or an imaginary ocular optical axis which is an extension of the ocular optical axis, and an external world optical axis of the light beam entering the image-pickup optical system from the external world to substantially coincide; the display means displaying image information periodically and intermittently, the image-pickup means recording image information from the external world periodically while the display means is not displaying image information, and the composite display apparatus is characterized by further including a change means capable of changing the display period of displaying image information on the display means and/or the image-pickup period of recording image information from the external world in the image-pickup means.
The above-described objects of the present invention can be accomplished with a head mounted display system using the composite display apparatus of the present invention.
The construction described above enables overcoming the disadvantages of the prior arts in that, when using the composite display apparatus in a light place, if the quantity of light from the display means, such as a liquid crystal display (LCD), is kept constant regardless of the increase in quantity of the see-through light guided from the external world directly to the eyes of an observer, the display tends to be faint relative to the see-through images, and if the shutter speed of cameras such as a CCD is kept constant, flare and smear may be caused on the picked-up image due to the excess light quantity, and contrarily, when using the composite display apparatus in a dark place, if the quantity of light from the display means, such as a liquid crystal display (LCD), is kept constant regardless of the decrease in quantity of the see-through light guided from the external world directly to the eyes of an observer, the display tends to be strong relative to the see-through images, and if the shutter speed of cameras such as a CCD is kept constant, the picked-up image may be dark or the contrast thereof may be decreased due to the insufficient light quantity.
Further, such a construction provides a composite display apparatus which enables keeping both optical density of display images formed by a display means and shutter speed of cameras such as a CCD in an optimum state relative to the see-through light, even in the use environments in which it is assumed that the quantity of see-through light varies widely. In addition, such a construction also provides a flexible composite display apparatus by allowing a composite display apparatus to be free from the constraint that the field period of the image-pickup system should correspond to the frame period of the display system.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.